Automated Detection of Vulnerabilities in Privileged Programs

We present a method for detecting exploitations of vul-nerabilities in privileged programs by monitoring their execution using audit trials, where the monitoring is with respect to speciications of the security-relevant behavior of the programs. Our work is motivated by the intrusion detection paradigm, but is an attempt to avoid ad hoc approaches to codifying misuse behavior. Our approach is based on the observation that although privileged programs can be exploited (due to errors) to cause security compromise in systems because of the privileges accorded to them, the intended behavior of privileged programs is, of course, limited and benign. The key, then is to specify the intended behavior (i.e., the program policy) and to detect any action by privileged program that is outside the intended behavior and that imperils security. We describe a program policy speciication language, which is based on simple predicate logic and regular expressions. In addition, we present spec-iications of privileged programs in Unix, and a prototype execution monitor for analyzing audit trails with respect to these speciications. The program policies are surprisingly concise and clear, and in addition, capable of detecting exploitations of known vulnerabilities in these programs. Although our work has been motivated by the known vul-nerabilities in Unix, we believe that by tightly restricting the behavior of all privileged programs, exploitations of unknown vulnerabilities can be detected. As a check on the speciications, work is in progress on verifying them with respect to an abstract security policy.